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I.  INTRODUCTION 


Research  for  the  improvement  of  combat-crew  training,  and 
the  efficient  execution  of  current  training  programs,  are 
heavily  dependent  upon  good  sources  of  information  about  trainee 
performance  during  and  at  the  end  of  training.  In  an  effort  to 
improve  training  performance  information,  this  study  is  directed 
to  (1)  systematic  definition  of  performance  and  (2)  development 
of  methods  for  measurement. 

This  program  is  divided  into  four  phases;  however,  the 
third  phase  has  been  divided  further  into  four  parts  as  the 
result  of  expansion  of  the  scope  of  the  program.  The  structure 
of  the  program  may  be  most  easily  comprehended  if  the  following 
planned  sequence  is  borne  in  mind:  (1)  establishment  of 

measurement  requirements,  (2)  establishment  of  measurement 
system  requirements,  (3)  conduct  of  design  studies,  (4)  develop- 
ment of  specifications  and  an  implementation  plan,  and  (5) 
preparation  of  the  Final  Report. 

As  shown  in  Figure  1,  seven  reports  will  be  prepared  under 
this  contract;  the  first  three  reports  prepared  all  present 
measurement  requirements  (Phase  I:  Pilot  Measurement  Require- 
ments; Phase  IIIA:  Combat-Crew  Measurement  Requirements; 

Phase  IIIB:  Air  Combat  Measurement  Requirements),  i.e.,  the 

measurement  to  provide  information  needed  for  combat-crew 
training  research.  These  requirements  have  been  determined 
through  surveys  conducted  at  combat-crew  training  sites.  The 
sites  visited  are  indicated  in  Table  1. 

The  current  Phase  II  Report  deals  with  the  requirements  for 
a measurement  system  to  process  the  measurement  which  has  been 
dictated  by  the  previous  reports . The  following  topics  are 
included  in  this  paper: 

1.  Research  procedures  to  indicate  operation  in  which 

a measurement  system  is  to  serve  as  a tool  in  achieving  research 
goals; 

2.  Measurement  processing  to  investigate  the  nature  of 
data  processing  associated  with  training  research  measurement; 

3.  System  criteria  to  guide  design  tradeoffs; 

4.  Preliminary  system  analyses  to  establish  measurement 
system  requirements  which  follow  rather  directly  from  the  system 
criteria . 
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TABLE  1 


COMBAT-CREW  TRAINING  SITES  VISITED  TO 
DETERMINE  MEASUREMENT  REQUIREMENTS 


Topic 

Site 

A- 7 

Luke  AFB 

F-4 

Davis-Monthan  AFB 

F- 106 

Tyndall  AFB 

B-52 

Castle  AFB 

C-141 

Altus  AFB 

C-130 

Dyess  AFB 

F-4E  Crew 

George  AFB 

C-141  Crew 

Norton  AFB 

Air  Combat 

Nellis  AFB 

... 


II.  RESEARCH  PROCEDURES 


The  topic  of  performance  measurement  is  best  not  considered 
in  isolation,  as  the  activities  associated  with  measurement  are 
integrated  into  all  the  major  activities  in  conducting  combat- 
crew  training  research.  In  this  context,  performance  measurement 
exists  only  to  achieve  specific  research  goals.  A thorough 
investigation  of  performance  measurement  system  requirements, 
then,  must  consider  the  characteristics  of  such  a system  in 
achieving  the  desired  research  goals. 

A typical  research  sequence  is  shown  in  Figure  2,  starting 
with  identification  of  a research  problem  and  terminating  with  a 
technical  report.  When  considering  assessment  of  all  factors 
affecting  measurement  system  design,  such  as  personnel,  facilities, 
equipment  and  procedures,  a meaningful  discussion  must  at  least 
consider  the  following  activities:  (1)  test  plan  and  measurement 

specification,  (2)  data  collection,  (3)  data  processing,  (4)  data 
analysis,  and  (5)  the  technical  report. 

Test  Plan.  The  test  plan  and  measurement  specification  is 
an  initial  statement  of  the  activities  of  the  research  crew  in 
achieving  specific  data  in  answer  to  the  research  problems  posed. 
All  of  the  specific  activities  shown  at  the  right-hand  side  of 
Figure  2 should  be  considered  in  this  statement.  It  is  important 
to  note  that  the  measurement  system  is  treated  here  as  a tool  for 
research  scientists  to  use  as  they  proceed  through  the  planned 
steps  in  accomplishing  a otudy;  it  is  perhaps  more  common  to 
consider  the  individuals  of  such  an. activity  as  supporting  the 
measurement  system,  but  this  is  a highly  nonproductive  approach. 

Data  Collection.  Research  data  must  be  meaningful  and 
consistent  (valid  and  reliable).  For  the  data  collected  to  lead 
to  meaningful  measurement,  the  research  crew  must  be  very 
familiar  with  expected  performance  and  the  difficulties  which  will 
be  met  in  achieving  criteria  performance;  that  is,  they  must  be 
as  knowledgable  in  the  task  requirements  as  the  combat  crews. 

This  knowledge  must  then  be  translated  into  performance  measure- 
ment to  produce  meaningful  quantification  of  combat-crew 
performance.  Consequently,  it  is  desired  to  guarantee  that  the 
research  crew  is  entirely  aware  of  the  nature  of  the  training 
mission  as  planned  and  accomplished,  and  all  factors  related  to 
performance  on  each  flight. 

Much  information  related  to  performance  measurement  can  be 
derived  from  the  mission  briefing,  monitoring  the  mission  (e.g., 
monitor  communications),  and  the  mission  debriefing.  In  some 
cases  there  is  so  much  information  related  to  desired  performance, 
mission  sequence,  surrounding  circumstances,  and  unexpected 
difficulties,  that  it  may  be  necessary  to  make  audio  recordings 
for  later  transcription. 


4 


Da ta  Processing.  Unless  very  similar  research  has  been 
conducted  often  before,  research  data  processing  .is  an  iterative 
process,  The  data  must  be  collected  and  inspected  to  see  if  the 
pJanned  measurement  is  appropriate  and  whether  more  meaningful 
measurement  forms  are  possible.  Alternative  measurement  schemes 
may  be  tried  out  with  some  of  the  initial  data  collected  to 
pick  the  best  forms.  This  implies  that  the  raw  data  are  stored 
in  some  form  which  permits  convenient  re-analysis. 

Unfortunately,  most  data  collection  systems  permit  frequent 
errors.  Bad  data  points  are  common.  Data  are  recorded  in  the 
wrong  sequence  and  are  mis-labeled,  or  require  identification  for 
later  automatic  retrieval.  While  it  is  possible  to  provide  much 
digital  computer  support  to  relieve  data  editing  chores,  data 
review  and  data  correction  are  ultimately  tasks  to  be  performed 
on  a case-by-case  basis  by  the  research  crew  in  a relatively 
manual  fashion. 

Data  Analysis.  While  some  statistical  treatments  of 
experimental  data  are  relatively  standard,  data  analysis  is  often 
a search  for  the  best  format  to  fully  present  the  experimental 
results.  Like  measurement  development,  data  analysis  is  an 
iterative  procedure.  The  research  crew  must  perform  trial 
analyses  with  early  data  collections  to  test  the  planned 
analytical  tools  (e.  g.  , analysis  of  variance,  multiple  range 
tests,  multiple  regression,  tabulations,  plots) . As  a result  of 
trial  analyses,  the  analytical  tools  may  be  changed,  or,  if 
unexpected  results  are  noted,  even  the  basic  study  may  be 
altered . 

Technical  Report.  The  goal  in  creating  a research  facility 
is  to  provide  an  environment,  facilities,  procedures  and 
responsibilities  to  allow  the  personnel  to  act  in  a closed-loop 
fashion  to  optimize  information  outputed  from  the  process.  If 
they  are  oriented  to  collecting  specific  data  and  publishing 
these  in  an  open- loop  fashion,  the  results  will  not  be  as 
desirable  if  the  research  output  is  directly  related  to  answering 
specific  research  questions.  In  general,  the  study  group  will 
tend  to  be  properly  oriented  and  professionally  associated  with 
their  work  output  if  they  are  responsible  for  a technical  report. 
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III.  MEASUREMENT  PROCESSING 


The  major  operations  in  data  processing  and  analysis,  aside 
from  those  involved  in  the  direct  data  collection  effort,  are 
shown  in  Figure  3 and  expanded  in  Figure  4.  The  major  operations 
are  discussed  in  the  following  paragraphs. 


Data  acquisition  and  data  control.  Acquisition  of  data 
requires  some  form  of  data  sampling  such  as  data  transmission 
and  recording.  As  data  control  may  involve  the  training  personnel, 
it  should  be  kept  simple;  however,  this  may  necessarily  include 
such  operations  as  turning  equipment  on  and  off,  setting  equipment 
modes,  and  possibly  marking  specific  events  for  measurement. 


Data  edit.  The  raw  data  should  be  stored  for  convenient 


playback;  each  data  block  must  be  distinctly  identified  to 
indicate  personnel,  equipment,  mission  and  training  conditions. 

It  is  likely  that  data  will  be  obtained  from  separate  sources, 
requiring  that  the  data  be  merged  and  synchronized;  for  example, 
data  could  be  recorded  inflight,  recorded  from  ground-based 
equipment,  and  collected  from  instructors  in  the  form  of 
subjective  judgments. 


Training  interface.  In  addition  to  providing  research 
information,  the  collection  of  performance  data  is  a valuable 
source  of  information  to  the  instructor  to  improve  training,  if 
such  information  can  be  presented  in  a readily  usable  form  for 
training  debriefings.  Further,  participation  of  instructors  in 
the  measurement  process  is  often  desirable.  Review  of  computed 
measures  with  the  instructors,  in  comparison  with  the  performance 
as  they  interpret  it,  can  be  a valuable  stimulus  to  improved 
measurement  development.  Post-mission  event  marking  may  be 
critical  to  achieving  some  forms  of  performance  measurement. 
Subjective  judgments  may  be  collected  for  correlation  with  other 
measurement.  However,  these  benefits  are  only  likely  to  occur 
if  there  is  a method  for  presenting  a summary  of  a given  flight 
so  that  specific  events  in  that  flight  can  be  marked  and 
discussed.  Whereas  data  may  be  encoded  in  some  form  convenient 
for  research  processing,  the  preceding  requirements  indicate 
collection,  or  collection/conversion,  in  a form  analogous  to  the 
aircraft  cockpit  presentation  so  that  performance  may  be  viewed 
in  a form  readily  interpreted  by  flying  personnel  as  well  as 
research  personnel. 


Measurement  calculations.  The  measurement  calculations 


chosen  of  course  will  depend  on  the  information  desired  and  the 
specific  procedures  developed  for  measurement.  To  calculate  a 
given  measure,  other  parameters  may  be  needed  in  addition  to  the 
primary  performance  parameters  (such  as  air  speed  or  heading)  to 
perform  the  calculation  (e.g.,  weight  off  the  wheels  to  indicate 
the  parameters  at  lift-off)  and  to  compare  to  desired  or  command 
performance.  • These  parameters  must  also  be  entered  into  the 
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Figure  4.  Research  Measurement  Processing. 
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system  as  required  for  the  type  of  maneuver  under  analysis.  The 
calculated  measures  form  a secondary  data  base;  that  is,  a data 
bank  is  gradually  accumulated  during  a study  upon  which  various 
analyses  may  be  performed.  This  data  bank  requires  care  and 
maintenance  much  like  the  raw  data  base,  and  additional  attention 
to  create  a file  structure  for  automatic  information  retrieval. 

Review  of  the  computed  measures  is  likely  to  reveal 
inadequacies  unless  the  measurement  has  been  previously  thoroughly 
tested.  Often  incorrect  assumptions  are  made  (perhaps  unknowingly) 
in  the  design  of  measurement;  or,  the  performance  measured  is  not 
precisely  that  desired.  It  is  therefore  extremely  important  to 
the  design  of  valid  and  reliable  measurement  (and  therefore  to 
the  success  of  the  entire  research  effort)  that  the  opportunity 
and  capability  for  trial-and-error  measurement  development  be 
permitted.  Unfortunately,  while  much  measurement  development 
can  take  place  prior  to  testing,  many  kinds  of  measurement 
problems  do  not  appear  until  tested  with  operational  data. 

Measurement  and  Research  Analysis  programming.  It  may  be 
seen  from  Figure  3 that  the  measurement  programming  and  research 
analysis  programming  operations  are  key  links  between  the 
research  personnel  and  the  tools  with  which  they  are  attempting 
to  achieve  the  research  goals.  This  link  must  be  kept  as  simple 
and  as  direct  as  possible.  It  is  therefore  desirable  that  the 
research  personnel  be  capable  of  performing  most  of  the  program- 
ming themselves,  although  the  need  for  an  experienced  systems 
programmer  may  also  be  justified. 

Since  combat-crew  training  research  studies  are  such  rich 
sources  of  information,  data  processing  and  analysis  generally 
entail  operations  with  large  volumes  of  data.  A digital  computer 
is  a modern  necessity  for  this  purpose.  For  the  types  of 
operations  necessary  for  research,  the  digital  computer  is  only 
useful  as  a tool  if  the  research  crew  is  capable  of  the  prepara- 
tion of  computer  programs  for  data  processing  and  analysis; 
further,  they  must  have  access  to  a computer  on  an  extensive 
interactive  basis.  BASIC,  ALGOL,  and  FORTRAN  are  languages 
which  permit  scientific  personnel  to  directly  communicate  with 
a digital  computer;  for  the  current  purposes,  the  FORTRAN  IV 
language  provides  commonality  to  other  research  and  utility 
programs,  ease  of  use,  and  power  for  scientific  computation. 

Research  processing.  As  previously  discussed,  selection  of 
the  proper  research  data  treatment  also  requires  an  iterative 
approach  similar  to  that  recommended  for  measurement  development. 
Consequently,  the  same  comments  about  availability  of  the  digital 
computer  and  programming  language  apply. 

System  operation.  From  the  foregoing,  it  can  be  concluded 
that  the  research  team  should  consist  of  research  scientists 
with  military  flying  knowledge  and  programming  skills  in  addition 
to  other  requirements  for  performing  empirical  research. 
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Programmers  may  be  required  depending  on  the  computational 
complexities  and  the  size  of  the  data  processing  facility. 
Engineers  and  technicians  will  be  needed  to  install,  repair, 
and  maintain  any  equipment.  Additionally,  since  such  data 
processing  inevitably  involves  many  manual  operations,  data 
clerks  and  secretarial  help  will  be  required. 

Combat-crew  training  research  may  take  place  at  any  of  a 
large  number  of  Air  Force  bases.  There  is  also  a distinct 
possibility  that  each  study  undertaken  may  occur  at  a different 
place.  The  research  facilities  at  most  training  bases  are 
rather  meager;  consequently,  the  facilities  must  be  collected, 
installed,  maintained,  repaired,  and  operated  at  each  site  as 
needed.  The  ability  to  locate  and  maintain  all  required 
ingredients  is  necessary  to  the  performance  of  timely  and 
successful  training  research. 


IV.  SYSTEM  CRITERIA 


System  criteria  are  needed  to  guide  design  of  an  appropriate 
measurement  system  and  to  provide  a basis  for  tradeoff  analyses 
of  candidate  systems.  The  criteria  generally  must  be  as  complex 
and  multidimensional  as  the  system  to  which  they  apply;  however, 
if  a simple  criterion  is  desired,  one  should  view  a measurement 
system  as  an  information  processing  system  and  ask  that  it  be 
feasible  and  usable,  but  most  importantly  that  it  provide  the 
information  needed  for  researdh. 

Criteria  for  a measurement  system  to  produce  needed  informa- 
tion for  combat-crew  training  research  are  listed  in  Table  2. 
Eleven  criterion  statements  are  made;  each  of  these  is  discussed 
in  the  following  paragraphs. 

Provide  needed  information.  During  visits  to  selected 
combat-crew  training  units  for  six  different  aircraft,  information 
requirements  for  use  during  such  training  were  identified.  The 
training  information  needs  were  summarized  in  the  form  of 
prototype  measurement  (these  are  to  be  found  in  Volume  II). 

Prototype  measurement  formats  identify  information  needed  during 
each  maneuver  of  each  training  phase,  representing  each  specific 
measurement  as  a blank  entry  after  a caption  indicating  the 
category  of  information  required.  The  format  thus  produced 
represents  a computer  printout  illustrating  the  type  of  measure- 
ment needed.  These  formats  are  quite  useful  for  the  development 
of  measurement;  however,  they  are  also  a useful  checklist  for 
evaluating  the  completeness  of  measurement  produced  by  candidate 
measurement  systems.  These  formats,  then,  provide  a means  for 
determining  whether  a given  system  provides  needed  information. 

The  prototype  measurement  will  lead  to  the  development  of 
detailed  measurement  definitions,  which  may  modify  the  informa- 
tion the  measurement  system  will  be  expected  to  produce. 

Further,  specific  research  may  pose  special  information 
requirements  not  found  from  examination  of  the  operational 
training  ne’eds;  and,  additional  information  needs  for  training 
may  be  discovered  in  the  future.  Consequently,  the  statement  of 
information  requirements  developed  in  this  study  must  not  be  used 
without  also  considering  needs  for  flexibility  and  growth 
potential . 

Provide  data  in  a useful  form.  While  the  current  measure- 
ment system  design  is  oriented  primarily  to  the  needs  of  training 
research,  it  is  conceivable  and  most  desirable  that  it  also  be  of 
value  for  operational  combat-crew  training.  It  is  desirable  that 
the  information  produced  is  readily  interpretable  by  instructor 
pilots  and  training  managers  to  (1)  provide  consultation  for  the 
development  of  better  measurement,  and  (2)  allow  pilot  inputs  to 
the  system  for  measurement . not  tractable  with  fully  automatic 
techniques.  One  may  conclude  then  that  information  should  be 
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TABLE  2 


RESEARCH  MEASUREMENT  SYSTEM  CRITERIA 


1.  Provide  needed  information. 

2.  Provide  data  in  a useful  form  for  (1)  combat' 
crew  training,  and  (2)  research  objectives. 

3.  Short  research  cycle  time. 

4.  Minimum  initial/sustaining  costs. 

5.  Minimum  data  distortion. 

6.  Compatible  with  different  training  devices. 

7.  Permit  iterative  design  of  valid  measurement 

8.  Minimum  interference  with  training  processes 

9.  Permit  correlation  with  data  from  external 
sources. 

10.  Minimum  space,  weight,  cooling  and  power 
requirements . 

11.  Effective  self-sufficient  personnel/facility 
configuration  capable  of  achieving  training 
research  objectives. 
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presented  in  a form  readily  interpreted  by  pilots,  i.e.,  in  a 
form  similar  to  that  encountered  while  in  flight.  On  the  other 
hand,  the  research  scientist  require  his  data  in  a numerical 
format  for  editing,  maintaining  accuracy,  and  for  further 
computation.  For  research  purposes  then  information  probably  is 
best  presented  in  tabular  or  plot  format.  Consequently,  both 
Research  Scientist  and  Instructor  Pilot  format  requirements  should 
be  considered  during  evaluation  of  alternative  candidate  measure- 
ment systems. 

Short  research  cycle  time.  The  information  produced  by  a 
measurement  system  will  be  fully  useful  only  if  it  is  timely. 

It  must  be  possible  to  begin  measurement  for  research  soon  after 
a research  problem  is  identified  or  the  results  may  be  too  late 
to  affect  key  decisions.  If  data  collected  are  to  be  useful  for 
training  debriefing  the  results  must  be  available  shortly  after 
a mission  is  ended  (less  than  an  hour) . If  the  data  collected 
are  to  be  reviewed  to  determine  whether  additional  data  collection 
is  necessary  to  replace  missing  or  incorrect  data,  the  data  must 
be  available  for  inspection  at  least  on  the  same  day.  Data 
reduction  and  analysis  of  the  complete  data  collected  during  a 
given  study  must  be  available  for  interpretation  within  a 
relatively  short  period  after  the  last  data  are  collected; 
otherwise,  specific  data  idiosyncrasies  are  forgotten  and  the 
results  are  not  timely.  Since  it  is  expected  that  combat-crew 
training  research  will  be  required  to  be  executed  according  to 
demanding  schedules,  the  ability  of  a measurement  system  to 
respond  quickly  to  research  needs  is  a very  important  consideration, 

Minimum  initial/sustaining  costs.  As  research  must  always  be 
performed  with  limited  budgets,  system  costs  may  be  the  primary 
evaluation  factor.  All  costs  must  be  considered.  Initial  costs 
include  modification  of  devices,  installation,  design  and  equip- 
ment purchases.  Sustaining  costs  include  personnel  salaries, 
facility  leases,  supplies  and  services.  These  costs  must  be 
traded  off  against  the  utility  of  the  information  derived. 

Minimum  data  distortion.  Any  real  information  processing 
system  changes  the  information  upon  which  it  operates  in  undesired 
ways.  In  a measurement  system,  loss  of  data  fidelity  leads  to 
invalid,  unreliable,  or  missing  data.  If  data  are  frequently 
missing  or  uninterpretable,  research  studies  can  become 
meaningless . 

Compatible  with  different  training  devices.  Training 
research  may  take  place  with  any  existing  or  proposed  training 
devices.  As  the  goal  of  the  research  may  be  to  evaluate  the 
effectiveness  of  a given  training  device,  it  will  be  necessary  to 
collect  identical  measures  for  comparison  in  each  training  device 
(including  probably  operational  aircraft).  A measurement  system 
for  training  research  will  therefore  be  required  to  operate  in 
operational  aircraft,  large  flight  simulator,  and  smaller  part- 
task  trainers. 


I 

M 

\ if 


Permit  iterative  design  of  valid  measurement.  Measurement 
design  can  proceed  in  a purely  analytical  vein  only  to  a point. 

The  measurement  designs  must  then  be  tried  out  with  real  data 
sampling  a variety  of  human  behavior  and  system  conditions. 

The  Research  Scientist  is  then  provided  with  data  which  he  can 
use  to  design  different  or  better  measurement.  The  measurement 
system  must  provide  the  capability  for  change  in  a relatively 
easy  and  quick  fashion.  If  such  iterations  in  measurement 
design  are  sufficiently  convenient,  valid  measurement  appropriate 
to  the  research  needs  may  soon  evolve;  without  capability  for 
iterative  measurement  design,  the  ability  to  properly  develop 
measures  will  be  surely  impaired. 

Minimum  interference  with  training  processes.  While  all 
measurement  in  some  way  affects  the  process  measured,  it  is  a 
goal  of  all  measurement  to  minimize  such  interference.  A 
measurement  system  for  training  research  should  be  inconspicuous 
requiring  little  or  no  attentirn  from  the  student  or  instructor. 

Permit  correlation  with  data  from  external  sources.  All 
research  data  may  not  emanate  from  the  same  source.  For  example, 
some  data  may  be  collected  inflight,  others  from  ground-based 
equipment,  and  still  others  from  instructor  judgments,  operational 
publications,  or  even  the  data  of  an  independent  study.  For  any 
given  study,  data  from  any  number  of  such  data  sources  may  be 
collected,  combined  and  correlated. 

Minimum  space,  weight,  cooling  and  power  requirements.  It 
is  probable  that  a measurement  system  for  combat-crew  training 
research  will  involve  measurement  of  performance  in  fighter- type 
aircraft.  In  such  a circumstance,  it  is  very  difficult  to  place 
any  additional  equipment  onboard,  and  it  will  be  more  difficult 
if  the  equipment  is  large,  heavy,  and  requires  extensive  cooling 
or  power  resources.  In  any  case,  for  any  application,  the  ease 
of  use  of  measurement  devices  will  be  enhanced  by  minimizing 
demands  for  physical  resources. 

Effective  self-sufficient  personnel/facility  configuration. 

A combination  of  people  and  tools  is  necessary  to  perform 
training  research.  The  major  tool  needed  is  a performance 
measurement  system.  The  proper  combination  of  ingredients  must 
all  be  brought  together.  All  types  of  people  and  equipment 
must  be  sufficiently  near  to  each  other,  and  not  scattered 
about  the  country  where  they  cannot  effectively  interact.  The 
proper  mix  is  also  important.  A variety  of  talents,  and  a 
mixture  of  manual  and  high-speed  computer  techniques  are  likely 
to  be  most  successful. 
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V.  PRELIMINARY  SYSTEM  ANALYSES 


While  detailed  design  studies  will  be  reported  later,  this 
section  presents  a number  of  preliminary  analyses.  First,  an 
initial  determination  of  parameters  needed  for  measurement  is 
made;  these  affect  the  size  and  complexity  of  the  data 
acquisition  system  needed  and  allow  estimation  of  the  volume  of 
data  to  be  processed.  Next,  some  initial  design  decisions  are 
made  so  that  later  design  studies  can  concentrate  on  remaining 
design  details:  (1)  the  basic  form  of  the  data  acquisition 

system  is  determined,  (2)  specific  data  processing  equipment 
alternatives  are  listed,  and  (3)  a general  personnel/facility 
configuration  is  discussed. 

Measurement  parameters.  Even  if  information  requirements 
are  fully  specified,  the  parameters  which  must  be  sensed  to 
provide  required  measurement  are  not  obvious  and  do  not  follow 
directly.  Figure  5 may  help  to  clarify  the  relationship.  In 
the  figure,  the  output  measures  (0)  correspond  to  the  information 
requirements.  All  steps  in  producing  each  measure  must  be  known 
before  all  the  parameters  which  must  be  sensed  can  be  determined. 

In  addition  to  basic  test  parameters  (M) , the  following  types 
of  parameters  may  be  needed:  (1)  parameters  needed  for  implement- 

ing logic  to  start  and  stop  measurement  computations,  (2)  informa- 
tion related  to  desired  performance,  and  (3)  error  information 
derived  from  the  difference  between  actual  and  desired  perform- 
ance. In  short,  given  output  measures  0,  to  determine  other 
parameters  which  must  be  sensed  (M,S,D,E),  it  is  necessary  to 
determine  the  logic  and  computations  to  be  used  in  measurement 
{i.e.,  the  measurement  algorithms) 

While  exact  determination  of  all  sensed  parameters  must 
therefore  await  full  development  of  measures,  candidate  measure- 
ment algorithms  were  devised  (for  later  review  and  expansion) . 

These  preliminary  algorithms  were  used  to  produce  the  list  of 
measurement  parameters  presented  in  Table  3.  This  list  can  be 
used  to  estimate  the  magnitude  of  data  acquisition  and  data 
processing  capability  required. 

It  may  be  seen  that  there  is  a rather  large  number  of 
parameters  to  be  sensed  if  one  is  to  design  a system  capable  of 
measuring  for  all  phases  of  training.  A total  of  approximately 
90  parameters  are  needed  for  all  phases  of  combat-crew  training 
based  on  this  initial  tabulation.  In  Table  4 the  estimated 
number  of  parameters  for  each  training  phase  is  listed  along  with 
note  of  parameters  which  may  pose  special  instrumentation  problems, 

These  data  indicate  that  a data  acquisition  system  may  be 
required  to  sense  approximately  90  parameters,  and  record  14-34 
parameters  during  a training  mission.  With  the  possible 
exception  of  stick  movement  parameters,  which  may  require 
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A.  AIRCRAFT  CONFIGURATION 

1.  Stab  Trim 

2.  Flaps 

3.  Gear  Posn. 

4.  Nose  Gear  Steer  Engage 

5.  Main  Gear  Contact 

6.  Drag  Chute  Out 

7.  Speed  Brakes  Out 

8.  Spoilers 

9.  Wheel  Brakes  On 

10.  Thrust  Reverse 

11.  Refuel  Probe  Engage 

B.  AIRCRAFT  PARAMETERS 


TABLE  3.  MEASUREMENT  PARAMETERS 
FIGURATION  E.  RELATIVE  POS 


RELATIVE  POSN.  (OTHER  A/C, 

TGT . OR  FIX) 

1.  Azimuth 

2.  Elev. 

3.  Range 

4.  Vc  ' 

5.  Aspect  Angle 

6.  Tgt.  Heading 

7.  Tgt.  MACH 

HUD/IR/RADAR/FIRE  CONTROL  SYS 

1.  Aim  Pt.  Desired 

2.  Aim  Pt. 


-2. 

Stick  (Pitch  & Roll) 

4. 

Fit.  Path  Marker 

3. 

Pitch 

5. 

Bomb  Fall  Line 

4. 

Pitch  Rate 

6 . 

Aim  Point  Error 

5. 

Ro.ll 

7. 

*Error  vs.  Aim  Pt 

6. 

Heading 

8 . 

Antenna  Elev. 

7. 

Turn  Rate 

9. 

Azimuth 

8. 

Slip 

10. 

Range 

9. 

Fuel  Quantity 

11. 

Range  Gate 

10. 

Fuel  Flow 

12. 

vc 

11. 

TIT,  EPR, . . . 

13. 

h Action  SW 

12. 

Throttle 

14  . 

Lock  SW 

13. 

IAS 

Lock  Mode 

14. 

MACH 

15. 

Doppler  Drift 

15. 

ALT  (MSL/FL) 

16. 

Doppler  G/S 

16. 

Radar  Alt. 

17. 

Weapon  Rel.  SW 

17. 

Vert.  Vel . 

18. 

Weapon  Rel. 

18. 

Ang.  of  Attack 

19. 

Dry  Run  Call 

19. 

G 

20. 

21. 

Missile  vs.  A/C 

FLIGHT  PROFILE  ERROR  DATA 

A Elev. 

A Azimuth 

1. 

2. 

Freq.:  VOR 

: Tacan 

22. 

23. 

A Range 
A V„ 

3. 

: UHF 

u v c 

4. 

: VHF 

G.  GROUND  SCORING 

5. 

To/From 

1. 

Impact  Pt. 

6 . 

Marker  Beacon 

2. 

*Bits 

7. 

Glide  Slope/Loc. 

3 . 

Foul 

Bearing 

Radial 


10. 

ARC (DME) 

11. 

ADI  Steer  Bars 

12. 

*ALT . 

13. 

Fix  Error 

1 

1 

14. 

X-Track  Dev. 

15. 

Along  Track  Dist 

* 

16. 

*Heading  Error 

1 

I 

17. 

*Drift  Angle 

| 

18. 

*IAS 

1 

| 

D.  TIME 

Xi 

& 

1. 

GMT 

TERRAIN  AVOID.  RADAR 
1^  Mode : Plan/Prof ile 

2.  Tgt  (x) 

'3.  Tgt  (y) 

4.  HRL 

5 . # Scans 

TANKER 

1.  Lights:  Up/Down 

2.  : Fore/Aft 

3.  Centerline  Dev. 


-May  be  computed  from  other  parameters 
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TABLE  4 

APPROXIMATE  NUMBER  OF  MEASUREMENT  PARAMETERS 
FOR  EACH  FLIGHT  PHASE 


FLIGHT  PHASE 

REAL-TIME 

MEAS. 

PARAMETERS 

SPECIAL  INTERFACE 
NEEDS 

T.O.,  Climb  & Level-Off 
Pattern,  Land  & Go-Around 

15 

20 

Stab  Trim 
Gear  Contact 
Nose  Steer. 
Wheel  Brks. 

Rwy,  Dist  & C/L 

Stand.  Inst.  Approach 

12 

j 

Pen,  Tacan  Approach 

14 

Bearing 

X-Track 

ILS  Approach 

17 

Stab  Trim 

* 

GCA  Approach 

20 

Stab  Trim 
Tacan  Freq. 
X-Track 

i I 

Formation 

13 

Stab,  Throttle, 
Stick  Position 

i ( 

Intercept 

34 

Miss.ile/Scope 

Position 

*, 

ACM  . 

25 

t K 

Air  Refueling 

16 

Stab  Trim,  Stick 
Lights,  Position 

1 \ 

Ground  Attack 

24 

Wpn  Rel,  Offset, 
X-Track,  Posn, 
Ground  Scoring 

Air  Drop  (Single-Ship) 

23 

Stick,  Throttle, 
X-Track,  Along  Trek 
Position,  Wpn  Rel. 

Air  Drop  Formation 

21 

Posn.,  X-Track, 
Throttle  Rate, 
Wheel  Brakes, 
Gear  Control 

| 

Terrain  Avoidance 

14 

Radar,  X-Track 

sampling  10-20  times  per  second,  parameter  sampling  should  not 
have  to  exceed  1-2  times  per  second. 

These  data  are  useful  for  initial  specification  of  candidate 
measurement  systems  for  design  tradeoff  studies. 

Basic  form  of  data  acquisition.  Three  basic  forms  of  data 
acquisition  have  been  considered;  these  are  (1)  telemetry, 

(2)  onboard  computation,  and  (3)  recording. 

Telemetry  involves  the  real-time  transmission  of  measurement 
parameters  from  an  aircraft  to  a ground  station  for  processing. 

The  research  team  is  permitted  to  monitor  and  closely  supervise 
data  acquisition,  and  to  process  data  prior  to  return  of  the 
aircraft.  Therefore,  measurement  may  be  calculated  for 
presentation  to  the  flight  crew  immediately  upon  their  return. 

A directional  tracking  antenna  may  be  required,  but  in  any  case, 
reception  will  be  limited  to  line-of-sight , restricting  the  range 
over  which  data  may  be  collected.  Further,  real- time^'data.  proces- 
sing of  a large  volume  of  data  requires  a large  high-speed  , 

computer,  and  measurement  algorithms  are  restricted  if  all 
calculation  must  proceed  as  data  are  acquired. 

Measurement  results  may  also  be  avail... ble  at  the  end  of  a 
flight  if  measures  "are  calculated  onboard  the  aircraft  during  the 
mission.  The  direct  calculation _of  measures  in  a completely 
automated  fashion  would  greatly  simplify,  the  subsequent  measure- 
ment and  analysis  processes.  A small  onboard  computer  can 
conceivably  accomplish  this  end,  but  it  would  require  efficient 
programming  by  experienced  programmers.  Research  personnel  would., 
have  to  operate  indirectly  through  these  programmers,  and  the 
nature  of  the  software  would  be  such  that  any  changes  would  be 
very  difficult.  Consequently,  it  would  be  inconvenient  to 
iteratively  develop  new  and  improved  measurement.  It  is 
concluded  that,  with  an  onboard  computer  for  measurement,  the 
research  personnel  would  be  restricted  in  scope  and  flexibility 
in  attempting  to  achieve  research  goals. 

The  remaining  data  acquisition  alternative  is  recording  with 
subsequent  replay  for  analysis.  Recording  may  take  several  forms; 

(1)  magnetic  tape  recording  of  sensed  electrical  signals, 

(2)  video  tape  recording  of  the  outputs  of  a vidicon  camera,  and 

(3)  photography.  In  any  case,  these  recordings  must  be  processed 
after  a flight  returns  and  the  recording  can  be  removed  from  the 
aircraft.  Recording  does  not  suffer  from  the  disadvantages  of 
the  other  alternatives,  however,  permitting  analyses  and 
re-analysis,  not  necessarily  in  real-time,  and  permits  trial-and- 
error  development  of  measurement. 

It  is  therefore  recommended  that  a form  of  data  recording 
be  used  for  combat-crew  training  measurement.  Subsequent  tradeoff 
analyses  will  be  conducted  among  alternative  systems  all  based 
on  data  recording.  This  assumption  is  not  duly  restructive,  since 
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forms  of  telemetry  or  onboard  computation  may  also  be  used  with 
recording;  that  is,  while  recording  is  the  selected  primary 
form  of  data  acquisition,  the  other  forms  are  not  totally 
excluded  from  future  consideration. 

Alternative  types  of  data  processing  equipment.  Four  types 
of  data  processing  facilities  are  identified  in  Table  5.  Digital 
computer  support  may  be  in  the  form  of  a dedicated  computer,  a 
time-share  computer,  or  a remote  computer.  Depending  on  the 
form  of  data  recording  selected,  specialized  conversion  and  play- 
back equipment  will  be  needed. 

TABLE  5 

DATA  PROCESSING  FACILITIES 


1.  Dedicated  Computer:  TTY,  Computer,  Card 

Reader,  Line  Printer,  Mag.  Tape  Units, 
Disc. 

2.  Instrumentation  Conversion  Equipment. 

3.  Time-Share  Computer. 

4.  computer  at  Remote  Location. 


Conversion  and  playback  eqcripment  of  course  are  a necessary 
part  of  the  data  acquisition  portion  of-^i^he  system.  These 
equipments  are  mentioned  here  for  completenSr^^s^  they  would 
reside  with  data  processing  equipment.  ConversioiT^guigment 
•varies  with  the  form  of  information  coding  (frequency  moulrlation^ 
pulse "duration  modulation,  pulse  code  modulation,  etc.);  there- 
fore, generalized  equipment  may  be  needed  if  data  may  be 
produced  with  various  forms-  of  airborne  equipment.  Otherwise, 
the  principal  tradeoff  is  between 'the  xemaining  three  forms  of 
digital  computer  facilities.  

Data  processing  facilities  dedicated  to  flight  performance -- - 
measurement  invariably  include  a small  general-purpose  digital 
computer  (16-32K  memory)  with  a teletype  terminal,  card  reader, 
line  printer,  magnetic  tape  units,  and  disc  storage.  For  small 
operations,  the  disc  storage  can  be  eliminated,  but  the  disc 
greatly  improves  capacity  and  speed  of  operation. 

A time-share  computer  facility  consists  of  a teletype 
terminal  connected  via  a telephone  network  to  a large  computer 
complex.  It  permits  use  of  a large  high-speed  computer  on  a 
pay-as-you-go  basis.  The  teletype  is  limited  in  the  amount  of 
input  and  output  data  which  may  be  processed.  It  is  therefore 
suitable  only  where  manual  data  entry  is  required,  such  as  with 
video  or  photographic  recording.  Nevertheless,  large  scale 
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analyses  may  be  performed  with  such  a system  with  high-volume 
output  on  fast  printers  at  the  computer  .complex  (and  mailed  to 
the  user) . 

Both  a dedicated  computer  and  a time-share  terminal  offer 
a tool  which  Research  Scientists  .may  interact  on  an  essentially 
unlimited  basis.  This  interaction  is  vital  to  measurement 
development  and  research  analysis.  As  this  level  of  interaction 
is  not  permitted  with  a remote  computer  facility,  this  option 
must  be  strongly  rejected.  One  may  attempt  to  collect  data, 
write  programs,  and  then  go  to  a remote  computer  facility  for 
data  reduction,  measurement  calculation  and  statistical  analyses; 
however,  this  approach  has  never  met  with  much  success  as  a 
permanent  mode  of  operation. 

Personnel/f acili ty  configuration.  Based  on  consideration 
of  the  design  criteria  presented  in  Table  2 and  past  operational 
experimental  efforts,  it  is  recommended  that  the  personnel/ 
facility  configuration  should  have  the  characteristics  listed  in 
Table  6.  Combat-crew  training  research  will  require  operation 
at  different  training  bases  on  short-notice,  implying  that  all 
needed  facilities  are  easily  established  at  each  new  location. 

If  large  and  complex  facilities  are  determined  to  be  necessary, 
these  should  be  housed  in  a transportable  enclosure.  The 
digital  computer  is  a basic  tool  for  this  type  of  data  processing, 
but  to  achieve  maximum  benefits  the  research  team  must  have 
ready  exclusive  access  to  the  computer  and  any  programmer 
personnel  which  may  be  required.  Of  course,  data  conversion  and 
playback  equipment  appropriate  to  the  type  of  recording  used  must 
also  be  housed  for  ready  access. 

As  the  measurements  collected  may  also  be  used  directly  in 
the  training  process,  and  since  measurement  review  by  Instructor 
rilots  may  be  necessary  in  the  course  of  development,  an  inter- 
face (i.e.,  display  or  cockpit  console)  will  be  needed  to 
display  information  interpretable  by  pilots. 

The  facilities  should  include  standard  office  space  and 
equipment,  facilities  for  test  and  repair  of  measurement  and 
computing  equipment,  and  storage.  Aircraft  communication 
equipment  will  permit  monitoring  actual  events  on  measurement 
flights  so  that  the  research  crew  know  the  circumstances 
surrounding  data  collection. 

The"ba-si.Q.. research  team  consists  of  Research  Scientists, 
data  clerks  and  secretary , engineers,  and  technicians.  Program- 
mers must  be  included  if  the -resultant  digital  computer  facility 
is  large,  and  in  the  initial  stages  of- establishing  the 
measurement  facility.  Of  course,  the  number -and.  type  of 
engineers  and  technicians  will  depend  on  the  specifjic^sq^ 
ultimately  specified  in  this  study.  Manual  data^d^erations'a-r 
inevitable  in  training  research,  posing  a requirement  for  data 
clerks.  A secretary  can  also  aid  in  manual  data  operations  as 
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PERSONNEL/FAC I LITY  CONFIGURATION 


Transportable  facility 

Direct  Access  to  Digital  Computer  and  Programmer 

Training  Interface 

Research  Scientists 

Data  Clerks  and  Secretary 

Engineer  and  Technicians 

Data  Conversion  and  Playback  Equipment 

Standard  Office  Space  and  Equipment 

Test  and  Repair  Facilities 

Storage 

Aircraft  Communications  Equipment 


well  as  transcribing  audio  recordings  and  preparation  of 
technical  reports.  Naturally  the  Research  Scientists  are  the 
key  personnel;  they  must,  at  a minimum,  possess  knowledge  and 
skills  in  the  following  areas;  (1)  training  psychology, 

(2)  field  research,  (3)  military  flying,  (4)  statistical 
analyses,  and  (5)  programming. 
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